Lydia: A System for Large-Scale News Analysis

Periodical publications represent a rich and recurrent source of knowledge on both current and historical events. The Lydia project seeks to build a relational model of people, places, and things through natural language processing of news sources and the statistical analysis of entity frequencies and co-locations. Lydia is still at a relatively early stage of development, but it is already producing interesting analysis of significant volumes of text. Indeed, we encourage the reader to visit our website (http://www.textmap.com) to see our analysis of recent news obtained from over 500 daily online news sources. Perhaps the most familiar news analysis system is Google News [1], which continually monitors 4,500 news sources. Applying state-of-the-art techniques in topic detection and tracking, they cluster articles by event, group these clusters into groups of articles about related events, and categorize each event into predetermined top-level categories, finally selecting a single representative article for each cluster. A notable academic project along these lines is Columbia University’s Newsblaster [2,4,8], which goes further in providing computer-generated summaries of the day’s news from the articles in a given cluster. Our analysis is quite different from this. We track the temporal and spatial distribution of the entities in the news: who is being talked about, by whom, when, and where? Section 2 more clearly describes the nature of the news analysis we provide, and presents some global analysis of articles by source and type to demonstrate the power of Lydia. Lydia is designed for high-speed analysis of online text. We seek to analyze thousands of curated text feeds daily. Lydia is capable of retrieving a daily newspaper like The New York Times and then analyzing the resulting stream of text in under one minute of computer time. We are capable of processing the entire 12 million abstracts of Medline/Pubmed in roughly two weeks on a single computer, covering virtually every paper of biological or medical interest published since the 1960’s. A block diagram of the Lydia processing pipeline appears in Figure 1. The major phases of our analysis are: